There arc advances in techniques for digitizing or compressing data such as video and audio that are originally analog. The merit of using digitized data is to enable the storage amount of a storage apparatus and a limited transmission band width to be fully utilized along with maintaining the quality of data to be stored or transmitted by handling any kind of data containing video, audio, characters and the like in almost the same way or by using compression techniques in storage or transmission of data, or to enable advanced techniques such as error correcting techniques and ciphering techniques to be easy to use. For this reason, for example, video is handled as discrete digital data and an arrangement of pixels having pixel values indicating a brightness or color signal.
There have been proposed an international standard technique, MPEG (Moving Picture Experts Group), for compression-coding digital data such as video, audio and text to be transmitted. Especially, MPEG4 processes digital data such as video, audio and character object by object instead of, for example, handling the whole image.
When specified objects in an image are handled separately from the whole image, the data, which have been separated into a plurality of objects in decoding compression-coded data, need to be synthesized and information for the superimposition in synthesizing the data is necessary. The information contains, for example, a two-valued signal indicating whether or not a pixel of an image is included in a specified object, or a multi-valued signal indicating to what degree the pixel occludes the background.
The information indicating the significance represents the shape of a specified object. Only the pixels that are decided to be significant according to the information indicating tho significance have influence on the picture quality of the image after being synthesized. In other words, the insignificant pixels have nothing to do with the picture quality, so that coding only significant pixels can improve the coding efficiency.
A method called by shape adaptive discrete cosine transform (SADCT) that orthogonally transforms only significant pixels is disclosed in IEEE Transactions on Circuits and Systems for Video Technology vol.5, No.1, February 1995. This method subject only significant pixels to orthogonal transform.
However, since compression by coding is executed according to spatial or temporal correlation in images, if insignificant pixels are also subjected to coding along with significant pixels, the coding efficiency is sometimes reduced. For example, when all significant pixels are black, if insignificant pixels are black, coding the significant pixels along with the insignificant pixels does not make the coding efficiency reduced, while if the insignificant pixels are white, the lowered correlation causes the coding efficiency to be reduced.
With the aforementioned SADCT, when an image to be processed has insignificant pixels in a region surrounded by significant pixels, correlation in the vertical and horizontal directions are lowered, thereby reducing the coding efficiency.
As described above, the pixel value of the insignificant pixel has some influence on the coding efficiency, despite that it has little influence on the picture quality of a regenerated image. Therefore, padding is carried out in which the pixel value of an insignificant pixel is replaced with a value which is obtained based on the pixel values of the neighbor significant pixels. With this padding, adjacent pixels have almost continuous values at the boundary between significant pixels and insignificant pixels and therefore the correlation between each pixel is higher than the significant region before padding, whereby the coding efficiency can be improved.
A prior art data padding apparatus in which the above-described padding is executed using digital data to be processed and significance information corresponding to the digital data is explained as follows. FIG. 49 (ISO/IEC JTC1/SC29/WG11 MPEG96/N1469 46 page) is a block diagram showing the configuration of the prior art data padding apparatus. As shown in the figure, the prior art data padding apparatus has a data replacing means 4901. The data replacing means 4901 generates padding data based on significant part of data using the data and the significance information corresponding to the data, and replaces insignificant part of the data with the padding data.
FIG. 50 is a flowchart showing a processing procedure of the data replacing means 4901 included in the prior art data padding apparatus. FIG. 51 is a diagram for explaining the padding by the prior art data padding apparatus. The operation in the padding by the data padding apparatus is described referring to FIGS. 49 to 51, as follows.
Original data S4901 and input significance information S4902 which are the inputs to the prior art data padding apparatus are shown in FIG. 51a). S4901 and S4902 are synchronized to be input to the data padding apparatus as shown in FIG. 49, keeping the correspondence shown in FIG. 51a).
In FIG. 51a), the numerals d1, d2, . . . each are component data which are discrete digital data. 8 pieces of the component data d1, d2, . . . , d8 constitute a processing unit data. The padding is executed for each processing unit. The numerals i1, i2, . . . are component insignificance information indicating of the significance of each component data, each of which make a one-to-one correspondence with the component data d1, d2, . . . , 8 pieces of component significance information i1, i2, . . . , i8 constitute processing unit significance information. i3 and i7 among the component significance information indicate `significant` while the others indicate `insignificant`. Accordingly, among the component data contained in the processing unit data in FIG. 51a), the prior art data padding apparatus finds d3 and d7 to be significant and the others to be insignificant from the processing unit significance information.
In a step 5001 in the flow shown in FIG. 50, component data and component significance information corresponding to the component data are input to the prior art data replacing means 4901. In a step 5002, using the component significance information, it is decided whether the component data is significant or not. When it is significant, a step 5003 is executed and the component data is held as padding data generating data. When it is insignificant, a step 5004 follows without executing the step 5003.
In the step 5004, it is decided whether the padding condition is satisfied or not. The data replacing means 4901 generates and holds edge information corresponding to the first component data or the last component data which constitutes the processing unit data. Further in the step 5004, the padding condition is decided to be satisfied when the two padding data generating data are held or when there is the edge information and one padding data generating data is held.
When the padding condition is satisfied, a step 5005 and below are executed while when the condition is not satisfied, the process goes back to the step 5001 where next component data and the corresponding component significance information are input.
When the step 5005 is executed, padding data is generated based on the one or two held padding data generating data. In a next step 5006, the component data to be processed is replaced with the padding data and the corresponding significance information indicates is changed to indicate `significant`. In a step 5007, data after padding (hereinafter referred to as padded data) and significance information after padding (hereinafter referred to as padded significance information) which have been processed in the step 5006 are output. In FIG. 49, padded data S4903 and padded significance information S4904 become the outputs of the data padding apparatus.
In case of data shown in FIG. 51, the process is as follows. Initially, in the step 5001 in FIG. 50, the component data d1 and the corresponding component information i1 are input to the data replacing means 4901. In decision in the step 5002, the result is to be `insignificant`, the process makes a transition to the step 5004 where the padding condition is decided. Since the component data d1 is the first component data of the processing unit data shown in FIG. 51a), the data replacing means 4901 holds the edge information corresponding to the input d1. So far until this stage, the edge information is held, but no padding data generating data is held, so that the padding condition is not satisfied and thus the process goes back to the step 5001.
The following component data d2 and the corresponding component significance information i2 are input, and the steps 5002 to 5004 are executed similarly to the foregoing, and the process goes back to the step 5001.
The following component data d3 and the corresponding component significance information i3 are input, and in decision in the step 5002, the result is to be `significant`, so that the step 5003 is executed and thus the component data d3 is held as padding data generating data. Thereafter, when decision in the next step 5004 is carried out, since the edge information and one padding data generating data are held, the padding condition is satisfied and thus the step 5005 and below are to be executed.
In the step 5005, the data replacing means 4901, when one padding data generating data alone is held therein, generates the same padding data as the padding data generating data. In this case, the same padding data p1 as d3 is generated. In addition, the data replacing means 4901, when only one padding data generating data is held therein, still holds the data after generating padding data using the padding data generating data. The data replacing means 4901 quits holding the edge information which has been held after generating the padding data. Accordingly, at this time, one padding data generating data alone is held.
In the next step 5006, the insignificant component data d1 and d2 are replaced with the padding data p1 and the component significance information i1 and i2 are changed from those indicating `insignificant` to those indicating `significant`. In the step 5007, the padded data arranged in the order `p1, p1, d3` and the padded significance information indicating `significant, significant, significant`, are output.
The step 5001 is executed again. The component significance information i4 corresponding to the following component data d4 is input, and the steps 5002 to 5004 are executed in a similar way when the component data d2 is input, and the process goes back to the step 5001 again. d5 and i5, and d6 and i6 are similarly processed.
The following component data d7 and the corresponding component significance information i7 are input, and in decision in the step 5002, the result is to be `significant`, so that the step 5003 is executed and thus the component data d7 is held as padding data generating data. Thereafter, when decision in the next step 5004 is carried out, since the previous padding data generating data d3 is held and thus two padding data generating data are held, the padding condition is satisfied and thus the step 5005 and below are to be executed.
In the step 5005, the data replacing means 4901, when two padding data generating data are held, generates padding data by averaging the two padding data generating data. In this case, data p2 for generating padding data is generated, which is (d3+d7)/2. Further, the data replacing means 4901, when two padding data generating data are held, quits holding the most previous of the two padding data generating data after generating padding data using the two padding data generating data. Accordingly in this case, d3 will not be held, but only d7 will be still held.
In the next step 5006, the insignificant component data d4 to d6 are replaced with the padding data p2, and the component significance information i4 to i6 are changed from those indicating `insignificant` to those indicating `significant`. In the step 5007, padded data `p2, p2, p2, d7` and padded significance information `significant, significant, significant, significant` are output.
The following component data d8 and the corresponding component significance information i8 are input. In decision in the step 5002 the data is decided to be `insignificant`, so that the process makes a transition to the step 5004 where the padding condition is decided. The component data d8 is the last component data of the processing unit data shown in FIG. 51a), so that the data replacing means 4901 holds the edge information. Accordingly, the edge information and one padding data generating data are held and thus the padding condition is satisfied, so that the step 5005 and below are to be executed.
In the step 5005, because of holding only one padding data generating data, the data replacing means 4901 generates the same padding data as the padding data generating data. Thus, the same padding data p3 as d7 is generated. In the next step 5006, the insignificant component data d6 is replaced with the padding data p3 and the component significance information i8 is changed from one indicating `insignificant` to one indicating `significant`. In the step 5007, padded data which is the padding data p3 and padded significance information indicating `significant` are output.
As described above, the processing unit data and the processing unit significance information shown in FIG. 51a) are subjected to the padding. FIG. 51b) is a diagram showing the result of the padding for the data. As shown in the figure, the insignificant component data contained the processing unit data shown in FIG. 51a) are replaced with the padding data generated based on the neighbor significant unit data, and the padded data are obtained.
For data before the padding, the insignificant component data d1, d2, . . . do not always approximate significant component data. For example, in case of video data, the component data d1, d2, . . . each are pixels, and the pixel values of each pixel which represents a brightness signal and a color signal do not always have the approximate values. As opposed to this, in the data after the padding, adjacent component data approximate each other, so that when the compression-coding processing is executed according to the correlation, the coding efficiency can be improved.
As hereinbefore pointed out, the prior art data padding apparatus subjects the input data and significance information to the padding, but as described above, the data padding apparatus works on the premise that the data and the significance information keep the correspondence with each other and are synchronized to be input.
In case that the compression-coded data is expansion-decoded before being used, in padding data, the orders of data and significance information does not always agree with time series or keep the correspondence with each other, depending on a coding method adopted in coding. When these data and significance information are to be processed, since they cannot be directly input to the prior art data padding apparatus to be subjected to the padding, an additional apparatus for synchronization is required so that data is input to the data padding apparatus after being subjected to the synchronization.
Further, for the prior art data padding apparatus, in case of complicated or advanced padding such as a case that data having a multi-dimensional structure is subjected to multi-dimensional padding, because great amount of data are stored and abundant calculation are carried out, a problem arises whereby the necessary amounts of storage media and the scales of circuits increase.
Furthermore, the prior art data padding apparatus repeats the processing procedure of the similar decision and process simply for each component data, so that there is still room for improvement in the coding efficiency.